Stepsize Selection for Approximate Value Iteration and a New Optimal Stepsize Rule

Approximate value iteration is used in dynamic programming when we use random observations to estimate the value of being in a state. These observations are smoothed to approximate the expected value function, leading to the problem of choosing a stepsize (the weight given to the most recent observation). A stepsize of 1/n is a common (and provably convergent) choice. However, we prove that it leads to very slow convergence for a basic instance of approximate value iteration, to the point of being unusable for most applications. We then derive an optimal stepsize rule for that instance. This is the first stepsize rule that explicitly takes into account the covariance between the current observation and the previous prediction in approximate dynamic programming. The rule can be easily extended to general ADP settings; experimental results show that it produces fast convergence without the need for tunable parameters.